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MIRANTIS 


OpenStack Bootcamp 


Organizational Notes 


We have 10 laptops. 

Laptop username/password: stack/stack 

Dashboard password(all users): nova 

VM instance username/password: cirros/cubswin:) 

The passwords have been applied to the footer of the document for your reference 
Checkpoints to pace the exercise and verify understanding of topics 


What version of OpenStack? 


We base our tutorial on Essex/Stable version of OpenStack. 

Most of the time we will be using an all-in-one installation (all components of OpenStack on a 
single laptop), so that everybody can do their exercises independently. For multi-node part, we 
will bring all the laptops together and they will be acting as a single OpenStack cluster. 


The installation was done using DevStack. To get an overview of DevStack, go to: http:// 
www.devstack.org 


Reference: 

Laptop username/password: stack/stack 
Dashboard password(all users): nova 

VM instance username/password: cirros/cubswin:) 


Itinerary 


During the exercises we will try to show the following: 

Day 1 

How to check OpenStack Installation Health 

How to use various types of interfaces (graphical, cmdline) 

Accessing VM instances 

Create and attach persistent storage to instances 

Glance image management 

How to create tenants and users using keystone 

Lunch 

Use vlan networking and dedicated tenant networks 

How to ensure instance security with security groups 

How to provide external access to instances with floating ips 

Break 

10. How to troubleshoot different OpenStack components (“break & fix” session) 
Day 2 

11. Set up multinode installation of OpenStack 

12. Swift Lab 


0}. GV ON 


© ON 
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Checking Installation Health 


OpenStack Components 

nova-manage service list 

#You should see a list of nova daemons along with smiley faces. 

ps -ef |grep -e glance -e keystone 

#You should see processes for additional OpenStack components like Keystone and Gance 


Platform Services 

rabbitmq: 

sudo service rabbitmg-server status 
mysql: 

sudo service mysql status 

KVM: 


sudo service libvirt-bin status 


#Since we are running a single-node installation, everything is located on a single laptop. 


Screen Sessions 

It’s important to note that DevStack creates a screen sessions with many windows. Each 
window shows the output from different OpenStack daemon. 

To connect the to screen sessions run the command below 


cd ~/devstack 
./rejoin-stack.sh #this script will attach you to the screen session 


#When on this screen you will see a list on the bottom referring the different windows. 
#To navigate these windows type: ‘ctrl-a’ followed by ” - to show window list. 


Spend some time getting comfortable navigating these windows as this will be used later to stop 
and start nova components 


Checkpoint 1: 
e Check health OpenStack components and platform services 
e Got familiar with the screen sessions and switching between them 
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Starting with OpenStack - User Perspective 


A typical work scenario when you are a new OpenStack user is as follows: 


You are given username and password 

You login to the interface (it can be either a web GUI or a CLI) 

You generate and register an ssh keypair. 

You download the private ssh key to your computer (web GUI will ask for it 

automatically). 

5. The public ssh key is kept by OpenStack. It will be injected into instances that you 
spawn to /root/.ssh/authorized_keys. Thus you will be able to login to instances via ssh 
with your private key. Important notice here: keep your private key backed up in a safe 
place, as you own the only copy of it. 

6. You spawn a number of instances. 

7. Once they are up, you can adjust security groups (firewall) to let the traffic you want 

(typically it will be ssh and ICMP first), add floating IPs etc. 


TON a 
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OpenStack Dashboard 


OpenStack Dashboard is a web UI interface to OpenStack. It supports majority of standard 
operations. For production use of OpenStack we recommend using CLI. 
1. Login to OpenStack Dashboard with a browser: 
a. http://172.15.0.</aptoonumber> 
b. user/pass: admin/nova 
c. Choose “demo” as your project. 
2. Try a basic workflow: 
a. Generate a SSH Key 
i. Click on Access & Security 
ii. Click Create Keypair 
iii. Give any name you want and click Create Keypair 
iv. A pem file will automatically download to /home/stack/Downloads 
1. *this file will be sued later to access the VM without a password 
b. Provision a VM instance using the Keypair 
i. Click on Images & Snapshots 
ii. Find image: cirros-0.3.0-x86_64-uec and click Launch 
iii. Give the instance a name. ex. VM1 
iv. Leave User Data empty for now 
v. Select tiny for Flavor 
vi. Select the Keypair created in previous step 
vii. Leave Instance count as 1 
viii. Leave Security Group as default 
ix. Click Launch 
3. Upon creation, pay attention to status, task and power state http://docs.openstack.org/ 


developer/nova/devref/vmstates.html#create-instance-states 
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VM Task Power 


Create Instance State State State 


hie > 

Compute.api.create_db_entry_for_new_instance ( Building ) 
h 5 A 

y fa a > 
Compute.manager._start_building | Building | 
\ J 

Compute.manager._allocate_network ( Building | 
Vv 7o N 
Compute.manager._prep_block_device Building | 
Compute.manager._spawn ( Building ) 
/ a N 

| Active | 

A y 


4. Click on Overview 
a. Now that you have an instance the overview tab will display metrics like usage 
and uptime 
5. VM instance actions: 
a. Click on Instances & Volumes 
b. Click on Action drop down next to your instance and make note of the options 
i. Edit, View Log, VNC, Snapshot, Pause, Suspend, Reboot, Terminate 


Checkpoint 2: 
e Navigate Dashboard tabs- Overview, Instances & Volumes, Images & Snapshots, 
Access & Security 
e Provision VM instances with Dashboard 
e VM instance actions 
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Nova CLI 


To log in to Nova CLI, supply credentials by setting environment variables: 


export OS _USERNAME=admin #username 
export OS_TENANT_NAME=demo #project name 
export OS_PASSWORD=nova #password 


export OS_AUTH_URL=http://127.0.0.1:5000/v2.@ HURL of the authentication(keyst.) 


TO save time you can type: 
/home/stack/source credrc.sh 


#credrc.sh contains all the variables mentioned above 


The main CLI command for OpenStack is “nova”: 
nova help #Display all available commands 
nova list #List running instances 
#Look for the instance you provisioned from dashboard 


Lets now provision a VM instance using nova CLI. 
1. Gather some information on available resources (images, flavors, ssh-keys): 
a. nova image-list #try to figure out why you see 3 “images” instead of just one 
which dashboard showed 
b. nova flavor-list #to get a list of available flavors 
cC. nova keypair-list #to get a list of available keypairs 
2. Boot the instance using the above info: 
a. nova boot --flavor <flavor_id> --image <image_id> --key_name 
<ssh_key_name> <server_name> 


3. Watch the VM transition from BUILD to ACTIVE 
a. watch -n 1 nova list 


Checkpoint 3: 
e Supply credentials for CLI 
e Provision VMs from command-line 
e Watch status of VM provisioning 
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Accessing VM instance 


After the instance status is “ACTIVE”, we can login via Dashboard and CLI 


1. Dashboard> Instances & Volumes 
a. Click on a VM instance that you created in the previous step 
b. Click the VNC tab 
c. Login with cirros/cubswin:) 
2. CLI> Return to terminal window as stack user 
a. cd /home/stack/Downloads 
b. chmod 600 <key.pem> 
c. ssh -i <key.pem> cirros@<ip of the VM> 
#by using the pem key, you should be able to login without a password 


The metadata service 

The metadata service is one of the components of OpenStack. It exposes some data about the 
instance itself (for example the public-key to be injected into authorized_keys, etc.). 

It runs on the link local address of 169.254.169.254. The instance can make use of this data in 
many ways. Try to figure out how you could use the information provided by metadata service to 
facilitate your cloud deployment. 


Once you are connected to the instance, run the command below to get some information on it 
from metadata server. Type this on the console: 
wget -q -O - http://169.254.169.254/latest/meta-data/ 


After retrieving the list try appending an item on the list to the command above: 
wget -q -O - http://169.254.169.254/latest/meta-data/hostname/ 


http://docs.openstack.org/trunk/openstack-compute/admin/content/metadata-service.html 
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Volumes - adding persistent storage 


Openstack creates virtual machines without any persistent storage as they are volatile can 
be terminated by users. This means that anything written to the VM OS hdd will be lost after 
terminating the instance. 


The answer for this is nova-volume, nova-volume uses any backend block storage and exposes 
it to the virtual machine for use 


Login to Horizon with admin/nova 
o Instances and Volumes tab 
o Create Volume 
o Give it any name, description 
o Size 3GB 
Delete the failed volume and Retry with Size 2GB 
o Check /opt/stack/nova-volumes-backing-file size to understand why 3GB failed 
Edit Attachments on the 2GB Volume and select either instance you’ve created 
o attach volume 
Verify that the volume is attached to the VM 
o ssh in to the VM that you attached the volume to 
o cat /proc/partitions 


Try detaching the volume and check /proc/partitions to verify that it’s no longer available. 


Checkpoint 4: 


SSH to VM with private key 


e VNC to VM using Dashboard 

e Get VM instance information from metadata service 

e Creating and attaching Volumes - Persistent Storage 

e Verifying persistent storage is available on VM 
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Glance - image management 


Glance is an image service for OpenStack. The default cirros image type is AMI (Amazon 
Machine Image) known from Amazon EC2. AMI images consist of three files: 

- kernel (AKI) 

- ramdisk (ARI) 

- filesystem (AMI) 


A binding between AKI, ARI, AMI is maintained by glance. So each time a user request an 
instance based on a AMI image, all these three components are inserted into the instance. 


Try to display what’s in glance now: 
glance index 


glance show <ami_id> #you should see the 3 images that comprise the cirros image 


Find the ubuntu image in /home/stack directory. The image type is qcow2 which means it’s a 
single image instead of three. 
precise-server-cloudimg-amd64-disk1.img 


Now upload this image to glance: 
glance add name=ubuntu_precise is public=true container_format=bare disk_format=qcow2 
< precise-server-cloudimg-amd64-disk1.img 


glance index #should list the new image 


Go to the dashboard and launch an instance with the new image. 
Try accessing this image using ssh (which you learned in a previous section) with ubuntu user. 


Checkpoint 5: 
e List images using CLI and Dashboard 
Understand AMI image format 
register a new image to Glance 
Use new image to run an instance 
Access instance based on uploaded image 
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Keystone - tenants and users 


Keystone is a component which provides identity, catalog, and policy services in OpenStack. 
You can compare it to Active Directory Services for Windows. Keystone is the place where we 
define and manage: - tenants - users - services - roles 

Up till now we were operating in a single-user environment, using only “demo” project. 

Time to add a new user and tenant. 

export OS_USERNAME=admin #switching to admin allows us to create tenants and users 


keystone help #display available options 


We define the following: 
1. A tenant 
a. keystone tenant-create --name bootcamp --description ‘bootcamp’ -- 


enabled true 
b. keystone tenant-list #show tenants 
cC. export TENANTID=<bootcamp tenant ID>#this will store tenant ID 
2. A user associated with the tenant 
a. keystone user-create --name bootcamp --tenant_id $TENANTID --pass nova - 
-email test@sample.com --enabled true 


b. keystone user-list #list users 
C. export USERID=<bootcamp user ID> #this will store user ID 
3. Grant “bootcamp” user “admin” role for the “bootcamp” tenant 
a. keystone role-list #display predefined roles, copy admin role id 


b. keystone user-role-add --user_id $USERID --role_id <admin_role_id> -- 
tenant_id $TENANTID 


Create a new instance with the bootcamp user: 

e Log in with the bootcamp user and launch a VM Instance 

e The VM should fail at the networking task step. 

e Lets also check what happened to the instance via CLI. Remember to export OS_* vars 
with the values of the new user and tenant otherwise you will be troubleshooting as the 
demo user. 

a. export OS _USERNAME=bootcamp 

b. nova list 

#error with invalid credentials because the tenant is still set to demo 

c. export OS_TENANT_NAME=bootcamp 

d. nova list #find your instance ID and copy it 

e. nova show <instance_id> #take a look at task_state, vm_state fields 

Checkpoint 6: 

e Create users and tenants with Keystone 
e Look for errors in Dashboard and CLI when instances fail to launch 
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VianNetworking 


This diagram illustrates vlan networking inside your laptop when you run a single-node 
OpenStack installation. For each tenant a dedicated bridge is created. Tenant's vm-s attach to 
this bridge. Places where 802.1q vian tags are applied to vm traffic are marked with green color. 
We can see, that within a single node installation, inter-tenant-network traffic is not tagged, but 
routed internally between tenant bridges. 

Interface ethO.105 is a “public” interface on which we connect to API. It is also the default 
gateway via which tenant vm-s reach the world. 


Laptop 


tenant1 tenant2 


br100 br102 
10.0.0.1 10.1.0.1 


(tenant1 gw) (tenant2 gw) 


i 
tan | etho.105 -viant02 


172.15.0.55 (public ip) 
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Introducing “nova-manage” command 


A new command has been introduced here called “nova-manage”. This command is used to 
perform administrative tasks on OpenStack, such as: 
e managing nova daemons 
e setting quotas 
e managing networks 
Use the following commands to see why the VM instances in bootcamp tenant are failing. 
keystone tenant-list 
nova-manage network list 
#look at the output and notice tenant id in the network we have now. 
#compare it with the new tenant’s id 


You can see that the only network that is present, has been created for tenant “demo” and not 
for “bootcamp” This is why the provision VM step was failing for bootcamp user. 


Use the following command to create a network for a new tenant: 
nova-manage network create --fixed_range v4=10.0.1.0/24 --vlan=102 -- 
project_id=$TENANTID --label=bootcamp 


Return to the Dashboard, delete the failed instance and create a new one. It should now be 
placed in the network you have just created. 


Verify that from the hypervisor (your laptop), you can reach vm-s on both tenants’ networks 
(just try to ping vm-s belonging to different tenants. This is because for each tenant net, the 
br<VLAN_NO> interface (e.g. br100) acts as a gateway, and is assigned the first address from 
the tenant’s dedicated network. 


To examine current network config, type on your laptop: 


brctl show #this will display bridge configuration 

sudo cat /proc/net/vlan/config #see where vlan tagging is done 

ifconfig br1ee #should be addressed with 10.0.0.1 

ifconfig br102 #should be addressed with 10.0.1.1 

ip route show|grep br1 #will show you routes to each of the tenant’s vians - that’s 


how we can reach them from the hypervisor 


Checkpoint 7: 
e Understand how network is configured to support multiple tenants 
e Create a second network for bootcamp tenant 
e Successfully create an instance with bootcamp tenant after network is created 
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Security groups 


At this point you should have instances running in both projects: demo and bootcamp. 
Log in to an instance of demo project and try to ping the instance from bootcamp. You 
see, there will be no response (even though you could ping both of them from the hypervisor!) 


Projects are isolated in terms of network connectivity. They are separated from world by so- 
called “Security Groups”. The purpose of security groups is to filter incoming traffic. Each tenant 
has at least one dedicated security group called “default”, in which all the tenant’s instances 
land if no other security group is specified for them. Tenants can create a number of security 
groups to address their specific needs (for example, you can have one group which lets ssh in 
and the other which lets only http). The security group for instance is determined on boot. To put 
an instance into a given security group, you need to pass an argument to “nova boot” command. 


So - to make “ping” work, you should explicitly let ICMP traffic in. Right now we have 

only “default” group defined. To modify default group - go to “Access & Security” tab in the 
dashboard, and edit group’s setting to allow ICMP (since ICMP is not aware of ports, you should 
give -1, -1 as the port range). 


How it looks under the hood 

Security groups manipulate iptables settings for instances on the hypervisor. For each newly 
spawned instance, a dedicated iptables chain is created. The rules are put into that chain - they 
reflect what is specified in the instance’s security group. To see it, go to root shell and type: 


sudo iptables -nL 


Look for nova-compute-inst-X chains - see if the ICMP rule is there. It will be there only for 
instances which belong to the secgroup you enabled ICMP for. 


Try to add ssh also. 
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Floating IPs 


In OpenStack we distinguish between two kinds of addresses: 

e fixed: these are allocated automatically to vm-s upon creation. You can compare them 
to an internal network in your datacenter, which is directly accessible to no one but your 
company. You already defined a fixed ip range with “nova-manage network create” 

e floating: these addresses are typically publicly routable. You can attach them to your 
vm-s, to bring them visible to the outside world. You can compare these addresses to 
your company’s public IPs. Typically in a datacenter you use your public IPs as follows: 

o you usually pay for public IPs, so you usually don’t want to afford to have as 
many of them as you have physical machines 

o you terminate them at a load balancer or firewall (the traffic is then passed further 
from the firewall to your datacenter host or load balanced to a set of datacenter 
hosts - in both cases private addressing is used) 


OpenStack follows this approach and distinguishes between fixed (private, go in multitude) and 
floating (public, relatively “scarce”). Fixed IPs are given to all booted instances in the cloud. 
Floating need to be taken from the pool by the tenant, and then attached to a certain instance. 
Also quotas on floating ips can be imposed by the administrator. 


This is a typical scenario on working with floating ip-s: 
OpenStack administrator registers a set of publicly available addresses in OpenStack 
(so-called “floating IP pool”). 
2. Tenant boots up his instances (they come up with fixed ips) 
3. Tenant grabs a floating ip from the floating IP pool 
4. Tenant associates floating ip to an instance. 


In dashboard, find the tab responsible for floating ip management. Try to add the ip to instance. 
Once you have a floating ip assigned to the instance, go to the terminal. 
Do two things: 
1. ssh to your instance using floating ip you just created (pay attention to enable ssh in 
security groups first). Type ifconfig inside your instance - what does it show? 
2. Go to the terminal on the laptop. Type two commands: 
a. sudo ip a 
b. sudo iptables -nL -t nat 
Find lines corresponding to your floating ip. 
Given all the above - how do you think floating ips work? 


Checkpoint 8: 
e Understand the role of security groups 
e Understand why we use Floating IPs 


Multi-node Setup 


We need to have one user running all central OpenStack services - we will call it the “controller 
node”. All other nodes will be compute nodes. 
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Cloud controller node 

Disable nova-compute service running on your laptop (use “nova-manage” command). 
nova-manage service disable --host=lab_<xx> --service=nova-compute 

confirm: 

nova-manage service list 


Attach yourself to devstack screen and turn off nova-compute and nova-network. 
/home/stack/devstack/rejoin_stack.sh | #Go to n-cpu and n-net and type Ctrl-C in each 


Detach from the devstack screen 
ctrl-a d 


Modify nova.conf file: 
sudo vim /etc/nova/nova. conf 
multi_host=true #add to the end of the file 


Attach yourself to devstack screen. Turn on nova-network(the window named “n-net”) 


Grab some coffee & wait ;-) 
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Compute nodes 
Terminate all your instances from OpenStack Dashboard Since it’s kvm under the hood, you 
can also check if you have running vm-s with “virsh list” as root. 


Attach yourself to devstack screen and turn off all the running OpenStack services: 
/home/stack/devstack/rejoin_stack.ss | #Go overall screen windows and type ctrl-c 


Go back to the terminal: 
sudo killall dnsmasg 


Dismantle all the networks that nova created for you: 


sudo ip link set down dev br1e@ #repeat this for every bridge you have 
sudo brctl delbr briee 
sudo iptables -F -t nat #flush everything that was left after running nova- 


network and nova-api 


Modify nova.conf file: 
sudo vim /etc/nova/nova. conf 
e Remove these lines if they are present: 
o flat_network_bridge=br100e 
o flat_interface=ethe 
e Change the following ($CC_IP == ip_of_cloud_controller_node): 
o sql_connection=mysql://root :nova@$CC_IP/nova?charset=utf8 
o rabbit_host=$CC_IP 
o glance_api_servers=$CC_IP:9292 
e Add this line to end 
o multi_host=true 


Attach yourself to devstack screen. Turn on nova-compute, nova-network, nova-api service (the 


window named “n-cpu”, “n-net”, “n-api”) 


Go to the CloudController guy and make him type: 
nova-manage service list #/s your node there? - what service is it running? 


At this point you also deserve some coffee, as you successfully made your laptop a part of the 
cloud! ;-) 
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Multi-node Setup - Check 


Now we try to boot new instances in the cluster we just created. We will use Dashboard for this. 
1. Connect to the Dashboard of the Controller node. http://<Controller_Node_|P> 
a. Login with admin user 
2. Spawn some instances in the demo tenant. The instances should land on the Compute 
nodes. 


Multiple tenants meet many nodes 


Create a new tenant and add users to it (you can reuse “bootcamp” tenant if you have it already 
configured. To display a list of users and tenants, use “keystone user-list” and “keystone tenant- 
list”). 

Try to bring up a dedicated network and instances for new tenant. See how network is 
configured on compute nodes. 

- check bridge configuration 

- see where iptables rules are put 

- see how vm interfaces are named 

- think about traffic flow between different compute nodes when two machines belonging to the 
same tenant need to exchange traffic 

- how do you think traffic between tenant networks flows 
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Swift Lab 


We base our Swift Lab on Essex/Stable version of OpenStack. 
The installation was done using SAIO-Swift All In One procedure. To get an overview of SAIO, 


go here: http://docs.openstack.org/developer/swift/development_saio.html 
The virtual machine emulates running a 4 node Swift Cluster. 


Getting Started 


Start VirtualBox 

Start>System>VirtualBox 

Choose SAIO> start 

SAIO OS User/Password: swift/swift 

startmain #this script will start the proxy, object, container, and account servers 


Swift CLI 

To log in to Swift CLI, supply credentials by setting environment variables: 
export ST_USER=test:tester #Username:Password 
export ST_KEY=testing #Key name 

export ST_AUTH=http://127.0.0.1:8080/v1.0 #URL of the authentication 
source swiftrc.sh #Alternatively, use swiftrc.sh to load the variables 


Get the status of swift installation 
swift --version #displays the swift version number 
swift stat #displays the status of the account 


Create a Container 


swift list #lists all available containers 

swift post bootcamp #creates a container named bootcamp 

swift list 

swift stat bootcamp #displays information about container: bootcamp 


Upload Objects 

cd /home/swift/tmp 

swift upload bootcamp test.mov #upload test.img to bootcamp container 
swift list bootcamp #lists all files in bootcamp container 

swift stat bootcamp test.mov #displays information about object:test.mov 


Try uploading test.img, test.doc, test.xls to bootcamp container on your own 
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Download Objects 

cd /home/swift/tmp 

rm test.mov #remove test.mov from local disk 

11 #verify test.mov has been removed 
swift download bootcamp test.mov #downloads test.mov to /home/swift/tmp 


Add metadata to Objects and Containers 
swift post -m Color:Blue bootcamp #adds metadata to container:bootcamp 
swift stat bootcamp #look for metadata that was added above 
swift post -m Genre:Comedy bootcamp test.mov 

#adds metadata to object:test.mov 
swift stat bootcamp test.mov #look for metadata that was added above 


Delete files in container 
swift delete bootcamp test.img 
swift list bootcamp #shows remaining file excluding test.img 


Delete Container(including all files) 
swift delete bootcamp #bootcamp container along with all files are deleted 


Authentication Tokens 

Obtain token: 

curl -v -H 'X-Storage-User:test:tester' -H 'X-Storage-Pass:testing' http:// 
127.0.0.1:8080/auth/v1.@ 

Use Token: 

curl -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url- 
above> 


Swift basic commands: 
http://docs.openstack.org/trunk/openstack-object-storage/admin/content/swift-cli-basics.html 


Checking Cluster Health 


There is a swift-dispersion-report tool for measuring overall cluster health. This is accomplished 
by checking if a set of deliberately distributed containers and objects are currently in their proper 
places within the cluster. 


First, populate the cluster with containers and objects with the command below: 
swift-dispersion-populate #This may take a few minutes 


Once the cluster is populated lets run the command below to check the health. 
swift-dispersion-report #Displays 100% for Containers and Object if the cluster is healthy 


http://docs.openstack.org/developer/swift/admin_ guide. html#cluster-health 
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